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BROADCASTING METHOD AND 



BROADCAST SIGNAL RECEIVING APPARATUS 



Technical Field 

The present invention relates to a broadcasting 
method and a broadcast signal receiving apparatus for 
presenting contents of video information such as a picture 
through a broadcasting network, more particularly, to a 
broadcasting method and a broadcast signal receiving 
apparatus wherein information presented through a 
broadcasting network and auxiliary information presented 
through other media such as package media can be watched and 
listened to in an interlocked manner. 

Background Art 

In the conventional system, edited video / audio 
information is broadcasted during a broadcasting band of 
time set as a program. Thus, the viewer can watch and listen 
to only part of such video / audio information actually 
broadcasted by a broadcasting station. For example, in the 
case of a news program, only part of gathered video 
information can be watched and listened to. In the case of 
a movie program, on the other hand, only part of a movie 
presented at a movie theater can be watched during the period 
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of the movie program usually shorter than the show time of 
the movie at the movie theater. 

Meanwhile, in recent years, there has been 
implemented a data broadcasting system whereby text data 
related to video / audio information is broadcasted as 
additional data to the video / audio information by 
appending the text data in the blanking signal of the video 
information. For example, systems called IT - vision and 
Intel - Intercast are known. In these data broadcasting 
systems, text data is described in a predetermined language 
such as an HTML (Hyper Text Markup Language) in a signal 
period of time of a video signal not used in video 
transmission to be broadcasted along with the video signal 
at the same time. An example of the signal period of time 
is an VBI (vertical blanking interval) . In the broadcast 
signal receiving apparatus, incoming video / audio 
information and text data are separated from each other . The 
video / audio information is played back as it is while the 
text data is displayed on a screen of a computer as a message 
related to the video information such as a text, a static 
picture or a graphic. In this way, the viewer is capable 
of not only watching and listening to the video / audio 
information but also obtaining information on its program 
at the same time. 



In the data broadcasting system whereby auxiliary 
information is broadcasted by using the VBI with a low 
information transmission capacity, however, there is a 
limit on the amount of auxiliary information that can be 
transmitted at a time, making it impossible to broadcast a 
large amount of data such as video information as auxiliary 
information. While video information and related 
information are broadcasted at the same time, they are 
independent of each other, thus, difficult to utilize 
related information in a process of synthesizing a video 
information such as a search for original information based 
on the related information and correlation of a scene cut 
out from a movie with the original movie . 

Disclosure of the Invention 

It is an object of the present invention to provide 
the viewer with video / audio data which can not be included 
in broadcast information so far as an auxiliary information 
by using a means adopting a broadcasting procedure different 
from the normal broadcasting procedure wherein the 
auxiliary information is synthesized automatically in the 
broadcast signal receiving apparatus with the broadcast 
information to generate final information which appears to 
the viewer as if the final information had been broadcasted 
as a continuous single program. 
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According to the present invention, broadcast 
information received by the broadcast signal receiving 
apparatus during a predetermined program band of time is 
played back in a manner interlocked with auxiliary 
information received in advance and held in the receiving 
apparatus . 

Brief Description of Drawings 

Embodiments of the present invention will be' 
described by referring to the following diagrams wherein: 

Fig. 1 schematically shows a package media 
interlocked broadcasting / watching / listening system; 

Fig. 2 shows the system configuration of a 1 st 
embodiment implementing a broadcast signal receiving 
apparatus; 

Fig. 3 shows the data format of broadcast information 
received by the 1 st embodiment; 

Fig. 4 shows the data format of auxiliary information 
received by the 1 st embodiment ; 

Fig. 5 shows a typical data format of an auxiliary 
information control table; 

Fig. 6 shows a typical video recording control table ; 

Fig. 7 shows a sequence of user operations; 
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Fig. 8 shows 2 different algorithms adopted by a video 
synthesizing unit for synthesizing broadcast information 
and auxiliary information; 

Fig. 9 shows the system configuration of a 2 nd 
embodiment implementing a broadcast signal receiving 
apparatus ; 

Fig. 10 shows the data format of broadcast information 
and auxiliary information received by the 2 nd embodiment; 
and ' 

Fig. 11 shows the data format of auxiliary information 
handled in a 3 rd embodiment. 

Best Mode for Carrying Out the Invention 

Fig. 1 explains the concept of a package media 
interlocked broadcasting / watching / listening system. In 
the system, contents of a broadcast program are created by 
a contents provider 900 and the contents are partly 
delivered to a broadcasting station 901 as broadcast 
information 201. The broadcasting station 901 broadcasts 
the broadcast information 201 through the infrastructure 
such as a satellite, a ground wave facility or a CATV. The 
contents provider 900 also distributes other information 
related to the broadcast information 201, through the 
Internet and package media, as auxiliary information 301. 
The viewer side (at a home 902) receives the broadcast 
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information 201 and the auxiliary information 3 01 through 
a broadcast signal receiving apparatus 100 to display the 
information on a monitor. In the package media interlocked 
broadcasting / watching / listening system, auxiliary 
information 301 supplements broadcast information 201. 
Both auxiliary information 301 and broadcast information 
201 include indexes 305 and 203 respectively for associating 
the former with the latter and for specifying how and with 
what timing the former supplements the latter. ■ 

The following is a description of an example to which 
the package media interlocked broadcasting / watching / 
listening system is applied. 

When distributing the contents of a large size theater 
movie, scenes in the movie are cut out to be the length of 
the broadcast information contents provider 900 adjusted to 
a broadcast time. The contents provider 900 delivers the 
movie -through- the-cutting-process to a broadcasting 
station, as broadcast information, to thereby be 
broadcasted to homes. On the other hand, the cut out scenes 
are sent to the homes as auxiliary information through other 
means such as the Internet, package media and data 
broadcasting systems. A broadcast signal receiving 
apparatus at each home receives the broadcast information 
and- the auxiliary information, thereafter, displays both 
the pieces of information on a monitor in the interlocked 



manner based on indexes appended to both. This enables the 
viewer to watch and listen to the contents of the complete 
movie. In a news program, part of news not broadcasted can 
also be presented to the viewer as auxiliary information. 
As a result, there is born a new information distribution 
business for distributing and delivering more detailed 
subjects, which are not broadcasted as a news program, as 
auxiliary information in the way described above. 

The package media interlocked broadcasting / watching 
/ listening system can also be applied to pair rental 
control. A pair rental control function prevents pictures 
and expressions of the original work inappropriate for 
juvenile viewers from being displayed. To be more specific, 
portions of the original work for general viewers are 
broadcasted as broadcast information whereas scenes for 
adult viewers are only delivered separately as auxiliary 
information by using typically package media. This allows 
the viewer to select whether or not such auxiliary 
information is to be displayed at a time to watch and listen 
to a broadcast program, implementing the pair rental control 
function to appropriately exclude such scenes from 
displayed information. Thus, chances that scenes 
inappropriate for juvenile viewers are inadvertently 
watched and listened to are eliminated. 
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Moreover, the package media interlocked broadcasting 
/ watching / listening system can also be applied to 
broadcast advertisements (CMs) . To put it in detail, by 
delivering a CM such as a product advertisement as auxiliary 
information, the CM can be played back in synchronization 
with a program properly. As a result, the number of 
opportunities in which a product advertisement is presented 
to the viewer in addition to CMs included in broadcast 
information can be increased. By setting an auxil'iary 
information CM time in addition to the traditional CM time, 
the broadcasting station is capable of increasing the 
advertisement revenue. In addition, the CM provider is 
capable of appending a CM relevant to a program to the 
program broadcasted by a broadcasting station that does not 
broadcast the CM. For example, an index of 'the name of a 
program' included in broadcast information can be used to 
play back an advertisement of a product such as a car 
appearing in the program from a CD-ROM with timing specified 
by the index. In this way, the CM provider is capable of 
letting the viewer watch and listen to a CM even if the 
broadcast station does not broadcast the CM. 

The following is a description of an application of 
computer graphics to the package media interlocked 
broadcasting / watching / listening system. If the playback 
apparatus includes a processor that is capable of executing 
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a program, auxiliary information played back in a manner 
interlocked with an operation to display broadcast 
information will not be limited to video and .audio data 
described above. For example, a CG program, a program 
displaying computer graphics (CG) , can be provided as 
auxiliary information. That is, a CG program can be 
delivered as auxiliary information by using package media. 
Invocation of a CG program is triggered by index information 
included in broadcast information. By providing a parameter 
required for a CG program from a broadcast information, a 
CG character moving synchronously with the broadcasting is 
let appear on a displayed screen of the broadcast 
information. Such an application is particularly effective 
for programs such as an animation one. The viewer is capable 
of carrying out operations such as replacing some characters 
appearing in an animation program with CG characters 
selected from package media. If the viewer is also capable 
of creating a CG character by itself, the viewer will be able 
to enjoy to see a CG character created by itself appearing 
and acting in a broadcasted program. Such a joy can not be 
realized by the conventional TV broadcasting. 

In the mean time, standardization including one known 
as MPEG4 is making progress in recent years. In the video 
and audio encoding field, a technology called content based 
encoding has been implemented. The content based encoding 
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is a technology wherein video materials (or objects) 
included in video data is encoded independently and, on the 
receiving side, the objects are decoded individually and 
finally synthesized before being displayed. In this 
technology, since only information on objects that change 
in the video data is transmitted, not only is it possible 
to considerably reduce the amount of information to be 
transmitted, but other merits can also be reaped by a video 
data producer. The other merits include a capability of 
creating entirely new video data by combination of pieces 
of video data of objects photographed individually. With 
this content based encoding used in transmission of 
broadcast information, if the package media interlocked 
broadcasting / watching / listening system using a broadcast 
signal receiving apparatus provided by the present 
invention is adopted, video data of some objects included 
in a broadcast signal can be replaced by video data of other 
objects recorded in a piece of exactly resembling package 
media . 

For example, video data of a 1 st person appearing in 
a program and video data of a 2 nd person other than the 1 st 
person are each subjected to content base encoding with the 
2 nd person closely imitating behaviors of the 1 st person. The 
encoded data of the 1 st person is broadcasted as part of 
broadcast information while the encoded video data of the 



2 n person is recorded on package media for distribution. 
If only the broadcast information is received and played 
back, the 1 st person will appear in a program as it is along 
with other objects such as a background. If the broadcast 
information is played back in a manner interlocked with the 
package media, on the other hand, the viewer will be capable 
of watching and listening to video data wherein the 1 st 
person is replaced by the 2 nd person. 

The following is a description of a broadcast 'signal 
receiving apparatus of the present invention which makes the 
package media interlocked broadcasting / watching / 
listening system described above feasible. 

Fig. 2 shows the system configuration of a 1 st 
embodiment implementing a broadcast signal receiving 
apparatus 100 provided by the present invention. As shown 
in the figure, the broadcast signal receiving apparatus 100 
comprises: 

a broadcast signal receiving means 101 for receiving 
broadcast information 201; 

a storage means 106 such as a hard disc drive for 
temporarily storing broadcast information 2 01 received by 
the broadcast signal receiving means; 

a removable media drive 104 for inputting auxiliary 
information 301; 

an input means 107 for inputting a user intention; 
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a processor 102 for carrying out processing to convert 
broadcast information 201 and auxiliary information 301 
into a video signal; 

a display unit 108 such as a color monitor for 
displaying information; 

• a display means 103 for converting video data 
generated by the processor 102 into a signal to be displayed 
on the display unit 108; and 

a memory unit 105 used for storing programs 'to be 
executed by the processor 102 and temporarily storing data 
in the course of signal processing carried out by the 
processor 102. 

While it is desirable to put a mountable as well 
as removable random - access medium such as a CD - ROM, a 
DVD and an MO in the removable media drive 104, a sequential 
medium such as a tape is also acceptable as well. 

Broadcast information 201 is stored in the storage 
means 106 in program units. When the viewer watches and 
listens to a program, broadcast information 201 stored in 
the storage means 106 is transferred to the processor 102 
which then carries out processing such as decoding on the 
broadcast information 201 by using the memory unit 105 as 
a work area. A result of the processing is then output to 
the display means 103 . If an index is extracted from the 
broadcast information 201 during the processing, auxiliary 
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information 301 indicated by the index is read out from a 
medium mounted on the removable media drive 104 . Processing 
is then carried out on the auxiliary information 301 and a 
result of the processing is output also to the display means 
103. 

It should be noted that means used for furnishing 
auxiliary information 3 01 are not limited to media such as 
a CD - ROM. Auxiliary information 301 can also be supplied 
to the broadcast signal receiving apparatus 100 by'way of 
a network such as the Internet or a data broadcasting system 
such as a satellite broadcasting system or a ground wave 
broadcasting system. In this case, auxiliary information 
301 received from a partner on the Internet or a broadcasting 
station of the data broadcasting system is transferred to 
the storage means 106 or the memory unit 105 shown in Fig. 
2 to be stored therein. In either case, the auxiliary 
information 301 stored in the recording means 106 or the 
memory unit 105 is played back in a manner interlocked with 
broadcast information 201. 

Fig. 3 shows the data format of broadcast information 
201 received by the 1 st embodiment. As shown in the figure, 
a plurality of program indexes 203 are inserted between 
pieces of video and audio data of each program. Each of the 
program indexes 203 includes information such as a program 
ID 204 for uniquely identifying a broadcast program, a 
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program name 205 given to the program, a start time 206 of 
the program, an end time 207 of the program and one or more 
auxiliary information IDs 208 each for identifying a piece 
of auxiliary information 301 to be interlocked during an 
operation to play back the program. In the case of an analog 
broadcast signal, a program index 2 03 is embedded in a VBI 
region of the video signal by using the ordinary NTSC system 
video signal. In the case of a digital broadcast signal, 
on the other hand, a program index 203 is broadcasted as a 
control packet other than the video signal. 

It is desirable to insert a plurality of program 
indexes 203 into a program. This is because, if a program 
index 203 is inserted into only the head of a program, no 
program index 2 03 can be acquired in a recording or playback 
operation of the program starting from a position after the 
only one program index 2 03 , making it impossible to play back 
the program in a mode interlocked with auxiliary information 
301 with the auxiliary information 301 used as a supplement 
to broadcast information 201. By inserting a plurality of 
program indexes 203 into a program, however, this problem 
can be solved. A program index 2 03 may include auxiliary 
information IDs 208 of all pieces of auxiliary information 
3 01 to be interlocked with the program or include auxiliary 
information IDs 208 of only pieces of auxiliary information 
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301 to be interlocked with the program starting from a point 
into which the program index 2 03 is inserted. 

Fig. 4 shows the data format of auxiliary information 
3 01 received by the 1 st embodiment. As shown in the figure, 
auxiliary information 301 comprises a plurality of 
individual auxiliary information pieces 304 and an 
auxiliary information control table 303 for centrally 
controlling the individual auxiliary information pieces 
304. Each of the individual auxiliary information 'pieces 
304 comprises audio / video data 306 and an auxiliary 
information index 305 including information for identifying 
a partner program to be supplemented by the video / audio 
data 306. The auxiliary information index 305 comprises an 
auxiliary information ID 3 07 for distinguishing the 
individual auxiliary information pieces 3 04 from each 
other, the title 308 of the individual auxiliary information 
piece 3 04, a partner program ID 309 for identifying a partner 
program to be supplemented by the video / audio data 3 06, 
a supplementation start time 310 with the beginning of a 
partner program to be supplemented taken as a base, a 
supplementation end time 311 and time length information 312 
of the individual auxiliary information piece 304. 

Fig. 5 shows a typical data format of the auxiliary 
information control table 303. As shown in the figure, the 
auxiliary information control table 303 is a single 
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collection of attributes of auxiliary information indexes 
305 of all individual auxiliary information pieces 304 and 
storage locations 401. That is, each line of the auxiliary 
information control table 303 comprises attributes of an 
auxiliary information index 305 and one storage location 
401. A storage location 401 is an address in a medium or 
a storage device at which the video / audio data 306 of the 
individual auxiliary information piece 304 is stored. The 
auxiliary information indexes 305 cataloged in thd 
auxiliary information control table 303 are sorted by 
partner ID 309. Note that it is desirable to automatically 
copy the auxiliary information table 303 into the storage 
means 306 at the time the medium storing the auxiliary 
information table 303 is mounted on the removable media 
drive 104. With the auxiliary information control table 303 
copied into the storage means 106, a search of the table 303 
for a desired piece of information can be carried out at a 
high speed. 

Next, a package media interlocked broadcasting / 
watching / listening method is explained. The package media 
interlocked broadcasting / watching / listening method 
comprises the steps of: 

carrying out a recording operation to record 
broadcast information 201 into the storage means 106; and 



carrying out a synthesis / playback operation to play 
back a program while supplementing the broadcast 
information 201 with individual auxiliary information 
pieces 304. 

First of all, the recording operation is explained by 
referring to Figs. 2 to 6 . The viewer records broadcast 
information 201 into the storage means 10 6 in the same way 
as a TV program is recorded into a VTR (video tape recorder) . 
In the case of a storage means 106 implemented by a digital 
information storage means such as a hard disc, an analog 
broadcast signal is converted into a digital signal such as 
an MPEG stream by a video capture in the broadcast signal 
receiving means 101 and the resulting digital signal is then 
stored in the storage means 106. The viewer is allowed to 
specify a recording time in the case of reserved recording 
or unattended recording, or carry out a manual operation to 
start and end the recording. The processor 102 receives and 
stores incoming broadcast information 2 01 in accordance 
with a request coming from a timer not shown in the figure 
in the case of unattended recording or a request made by the 
viewer in the case of manual recording. At that time, the 
broadcast signal receiving apparatus 100 catalogs the 
recorded program into a video recording control table 501, 
a table for cataloging recorded programs, like one shown in 
Fig. 6. As shown in the figure, each line in the video 
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recording table 201 comprises a program ID. 204 as well as 
a program name 205 extracted from a program index 2 03 of the 
broadcast information 201, a recording start time 504, a 
recording end time 505 and a recording location 506 or an 
address in the storage means 106 indicating the beginning 
of an area for storing the broadcast information 201. It 
should be noted that the recording start time 504 and the 
recording end time 505 shown in the video recording control 
table 501 are times with a program start time 206 'of the 
recorded program taken as a reference. That is to say, the 
recording start time 504 and the recording end time 505 are 
obtained by subtracting the program start time 206 from the 
time the recording is actually started and the time the 
recording is actually ended respectively. 

Next, a typical synthesis / playback operation is 
explained by referring to Figs, 2 to 7 . In order to obtain 
individual auxiliary information pieces 304 to be used for 
supplementing a program, a recording medium containing 
auxiliary information 3 01 is mounted on the removable media 
drive 104 . When the recording operation described above is 
not carried out, the broadcast signal receiving apparatus 
100 carries out a through operation wherein the processor 
102 processes incoming broadcast information 201 and 
displays the processed broadcast information on the display 
unit 108 by way of the display means 103 as is the case with 
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an operation to receive an ordinary television broadcast 
program. For control purposes, the processor 102 displays 
a playback button 602 on the screen of the display unit 108 
as shown in Fig. 7A. 

When the viewer selects the playback button 602 by 
using a cursor 601 or presses a playback request button 
provided on the input means 107 such as a remote controller, 
the broadcast signal receiving apparatus 100 displays a 
recorded program menu 603 created at that time from the video 
recording control table 501 on the display unit 108 in order 
to let the viewer select a program recorded in the storage 
unit 106 to be played back. The viewer then selects a 
desired recorded program from the recorded program menu 603 
as shown in Fig. 7B by specifying the program name thereof. 
As the name of the desired recorded program is specified, 
the video recording control table 501 is searched for the 
name 205 of the desired program and the program ID 204 
thereof is acquired. 

Next, the processor 102 forms a decision as to whether 
or not auxiliary information 201 can be acquired from a 
recording medium mounted on the removable media driver 104 . 
The decision is formed by determining whether or not the 
program ID 2 04 of the selected program acquired from the 
video recording control table 501 exits as a partner program 
ID 309 in the auxiliary information control table 303 which 
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was copied from the recording medium into the storage means 
10 6 when the recording medium was mounted on the removable 
media drive 104 for the first time. If the outcome of the 
decision indicates that auxiliary information 201 can not 
be acquired, that is, if no partner program ID 309 in the 
auxiliary information control table 303 matches the program 
ID 204, supplementation of the program by auxiliary 
information 201 is ignored^ The processor 102 then searches 
the video recording control table 501 using the program ID 
204 as a key for the recording location 506. Subsequently, 
the processor 102 reads out the program from the location 
506 to play back it . If the outcome of the decision indicates 
that auxiliary information 201 can be acquired, that is, if 
a partner program ID 309 in the auxiliary information 
control table 303 matches the program ID 204, on the other 
hand, the matching partner program ID 309 is used for 
fetching attributes 307 to 312 and storage locations 401 of 
each auxiliary information index 305 associated with the 
desired program to be played back from the auxiliary 
information control table 303. An auxiliary information 
menu 604 shown in Fig. 7C is then displayed on the screen. 
It should be noted more than one auxiliary information index 
3 05 in the auxiliary information control table 3 03 may have 
a partner program ID 3 09 matching the program ID 204, that 
is, more than one individual auxiliary information piece 304 



may be associated with the desired program. As shown in Fig . 
7C, the auxiliary information menu 604 includes a table of 
individual auxiliary information pieces 304 associated with 
the desired program and a 'Select all' button which is 
operated if it is desired to specify all the individual 
auxiliary information pieces 304. Beneath the auxiliary 
information menu 604, the following buttons 605 to 607 are 
also displayed on the screen shown in Fig. 7C. The 'Play 
back auxiliary information' button 605 is selected if it is 
desired to play back only the individual auxiliary 
information pieces 304. On the other hand, the 'Playback 
program' button 606 is selected if it is desired to play back 
the broadcast information 201 only, ignoring the individual 
auxiliary information pieces 304. Finally, the 'Synthesis 
/ Playback' button 607 is selected if it is desired to play 
back the program by supplementing the broadcast information 
201 of the program with the individual auxiliary information 
pieces 304. 

Assume that the viewer selects an 'auxiliary 
information 1' item representing one of the individual 
auxiliary information pieces 304 listed on the auxiliary 
information menu 604 and the 'Synthesis / Playback' button 
607. In this case, the processor 102 searches the video 
recording control table 501 using the program ID 204 as a 
key for a recording location 506, a recording start time 504 
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and a recording end time 505 of the specific broadcast 
information 201 to be played back. Then, the processor 102 
searches the auxiliary information control table 303 for a 
recording location 401 of the individual auxiliary 
information piece 304 corresponding to the 'auxiliary 
information 1' selected from the auxiliary information menu 
604 to be synthesized and played back with the specific 
broadcast information 201, a supplementation start time 310 
and a supplementation end time 311 of the individual 
auxiliary information piece 304 used in the synthesized 
playback. 

The recording start time 504 of the specific broadcast 
information 2 01 to be played back is compared with the 
supplementation start time 310 of the individual auxiliary 
information piece 304 to be synthesized. Since the 
recording start time 504 and the synthesis start time 310 
are both given with the program start time taken as a 
reference, the comparison can be decided to be justifiable. 
Video and audio data of the information with the smaller 
start time is played back as shown in Fig. 7D. 

An individual auxiliary information piece 304 is 
played back as follows. The processor 102 compares the 
supplementation start time 310 of the individual auxiliary 
information piece 304 with the contents of a timer also 
taking the program start time as a reference all the time. 
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As the contents of the timer become equal to the 
supplementation start time 310, the operation to play back 
the broadcast information 2 01 is temporarily suspended and 
an operation to play back the individual auxiliary 
information piece 304 is started. As the operation to play 
back the individual auxiliary information piece 304 is 
finished, the operation to play back the broadcast 
information 201 is resumed beginning with a portion at the 
supplementation start time 310 or the supplementation end 
time 311 of the individual auxiliary information piece 304. 
If the operation to play back the broadcast information 201 
is resumed beginning with a portion at the supplementation 
start time 310, the individual auxiliary information piece 
304 is merely inserted into the broadcast information 201. 
A method of synthesis to insert an individual auxiliary 
information piece 304 into broadcast information 201 is 
referred to as an insertion type synthesis method. If the 
operation to play back the broadcast information 201 is 
resumed beginning with a portion at the supplementation end 
time 311, on the other hand, the individual auxiliary 
information piece 304 replaces part of the broadcast 
information 201 between the supplementation start time 310 
and the supplementation end time 311 . A method of synthesis 
to replace part of broadcast information 201 with an 
individual auxiliary information piece 304 is referred to 



• # 

-24- 



as an replacement type synthesis method. The insertion type 
synthesis method and the replacement type synthesis method 
will be exemplified in concrete terms later. 

If the 'Play back auxiliary information' button 605 
shown in Fig. 7C is selected by the user, only individual 
auxiliary information pieces 304 are played back 
sequentially one piece after another beginning with one with 
a smallest supplementation start time 310 shown in the 
auxiliary information control table 303. If the 'Play back 
program* button 606 shown in Fig. 7C is selected by the user, 
on the other hand, only the broadcast information 201 
selected from the menu shown in Fig. 7B is played back. In 
addition, interface buttons 608 resembling buttons on a 
remote controller used for controlling an ordinary VTR are 
displayed on the screen as shown in Fig. 7D. The interface 
buttons 608 allow the user to specify operations to start, 
halt, temporarily halt, run fast and rewind the playback of 
the video data. When the user specifies the operation to 
halt the operation by operating one of the interface buttons 
608, the broadcast signal receiving apparatus 100 
terminates the operation to play back video data and resets 
the timer. 

In the above description, there is assumed a method 
whereby broadcast information 201 is recorded into the 
storage means 106 shown in Fig. 2 in program units and, in 
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an operation to play back a program, broadcast information 
201 of the program is synthesized with auxiliary information 
301. If the read / write speed of a hard disc drive used 
as the storage means 10 6 is not high enough to keep up with 
the recording and playback operations, however, part of the 
memory unit 105 can be used as a buffer between the processor 
102 and the storage means 106. The buffer allows the 
operations to record broadcast and play back information 201 
into and from the storage means 10 6 to appear as if they were 
carried out concurrently. As a result, the package media 
interlocked broadcasting / watching / listening method can 
be applied to a program which is currently being 
broadcasted. 

The buffer cited above works as follows. Broadcast 
information 201 received by the broadcast signal receiving 
means 101 is stored in a 1 st buffer area in the memory unit 
105. As the amount of stored broadcast information 201 
reaches a predetermined value, the broadcast information 
201 is transferred to the storage means 106 at one time as 
a batch. While the transfer is being carried out, the 
operation to store broadcast information 201 received by the 
broadcast signal receiving means 101 can be continued. In 
the mean time, broadcast information 2 01 transferred to and 
recorded in the storage means 106 is transferred back to a 
2 nd buffer area in the memory unit 105 in appropriate units 
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to be played back. Pieces of broadcast information 201 in 
the 2 nd buffer area are sequentially processed one piece 
after another by the processor 102 and output to the display 
means 103. In this way, while currently broadcasted 
broadcast information 201 is being recorded in the storage 
means 106, a playback operation can be continued at a time 
delay to a certain degree. While such simultaneous 
recording and playback operations are being carried out in 
this way, auxiliary information 301 can be used as' a 
supplement in accordance with program indexes included in 
the broadcast information 201 being played back. Time 
delays are resulted in due to supplementation by using 
auxiliary information 301. However, such time delays can 
be absorbed by the simultaneous recording and playback 
operations. If the insertion type synthesis method 
described above is taken as a technique of supplementation, 
the time delay between broadcast information being recorded 
and be information 201 being played back is widened. If the 
capacity of the storage means 106 is larger than a size 
corresponding to the time delay, however, there will be no 
missed information at all. 

Fig. 8 shows two different algorithms adopted in a 
synthesis of broadcast information 201 and auxiliary 
information 301. To be more specific, Fig. 8A is a diagram 
showing the insertion type synthesis method and Fig. 8B 



-27- 



shows the replacement type synthesis method. As shown in 
Fig. 8A, in the case of the insertion type synthesis method, 
the broadcast signal receiving unit 100 plays back auxiliary 
information 1 denoted by reference numeral 306 after playing 
back video and audio data 202 of broadcast information 201 
denoted by notation Ml till a point of time Tl . After the 
operation to play back auxiliary information 1 is finished, 
the operation to play back the video and audio data 2 02 of 
the broadcast information 201 is resumed starting 'with a 
portion M2 at the point of time Tl . In this way, auxiliary 
information 1 is merely inserted at the point of time Tl into 
the video and audio data 202 of the original broadcast 
information 201. In the case of the replacement type 
synthesis method shown in Fig. 8B, on the other hand, as the 
operation to playback the auxiliary information 1 is 
finished at a point of time T2 ; the operation to play back 
the video and audio data 2 02 of the broadcast information 
201 is resumed starting with a portion M3 at the point of 
time T2 . In this way, auxiliary information 1 replaces the 
portion M2 between the points of time Tl and T2 of the video 
and audio data 202 of the original broadcast information 
201. 

It should be noted that, in an effort made in the 
actual implementation to adjust timing in a playback 
operation, auxiliary information 301 is read out earlier 
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than an accurate start time and stored in a cache memory. 
By doing so, the auxiliary information 3 01 can be played back 
with good timing without causing the viewer to feel a sense 
of incompatibility. In addition, if interframe encoding is 
carried out to produce an MPEG stream, a frame on an encoding 
boundary, that is, an I frame in the case of MPEG encoding, 
is adjusted to coincide with a boundary between broadcast 
information 201 and auxiliary information 301. 

Next, a 2 nd embodiment implementing a broadcast' signal 
receiving apparatus provided by the present invention is 
explained. In the case of the 2 nd embodiment, auxiliary 
information 301 is also transmitted in the same way as 
broadcast information 201 instead of being delivered by 
using a recording medium. Fig. 9 shows the system 
configuration of the 2 nd embodiment. Since, auxiliary 
information 301 is also transmitted as is the case with 
broadcast information 2 01, as shown in the figure, the 2 nd 
embodiment is obtained by omitting the removable media drive 
104 from the system configuration of the 1 st embodiment shown 
in Fig. 2. Instead, an information separating program 801 
stored in the memory unit 105 is used for separating 
auxiliary information 301 from broadcast information 201 
received by the broadcast signal receiving means 101. A 
method for acquiring auxiliary information 301 in an 
operation to record video data is also different from that 
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adopted in the 1 st embodiment implementing the broadcast 
signal receiving apparatus 100 shown in Fig 2. 

An operation to record video data carried out in the 
2 nd embodiment is explained as follows . The broadcast signal 
receiving apparatus 100 receives broadcast information 201 
and auxiliary information 3 01 relevant thereto transmitted 
by a broadcasting station. Fig. 10 shows the data format 
of the broadcast information 201 and the auxiliary 
information 201 received by the 2 nd embodiment. The 
auxiliary information 3 01 is embedded in a program index 
203. The processor 102 executes the program 801 to extract 
auxiliary information 301 from information received by the 
broadcast signal receiving means 101 and store the 
information in the storage means 106. 

It should be noted that, in some cases, a large amount 
of auxiliary information 301 can not be transmitted by 
including the auxiliary information 301 in a program index 
203 of broadcast information 201as shown in Fig. 10 due to 
broadcasting specifications. In such a case, auxiliary 
information 301 is transmitted by splitting it into a 
plurality of packets. Typically, such packets are 
transmitted in advance independently of the associated 
broadcast information 201 prior to the time band of the 
program of the broadcast information 201. The processor 
102 employed in the broadcast signal receiving apparatus 100 
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stores auxiliary information 301 included in each packet. 
As the last packet is received, pieces of auxiliary 
information 301 are arranged in an order of sequence numbers 
included in the packets to reconstruct the entire auxiliary 
information 301. The 2 nd embodiment allows the viewer to 
automatically acquire auxiliary information 301 by 
receiving a broadcast signal without the need to resort to 
another means such as a recording medium used in the 1 st 
embodiment for distributing auxiliary information ' 3 01 . 

Next, a 3 rd embodiment of the present invention is 
explained. In the case of the 3 rd embodiment, the contents 
of auxiliary information 301 are a CG program, a sentence 
written in the HTML or an operation described as a script. 
In one typical version of the 3 rd embodiment, link 
information described in the HTML is received as auxiliary 
information 301. Video data generated by a source on the 
Internet is then acquired by tracing the link and displayed 
by executing a browser program. In another typical version, 
of the 3 rd embodiment, a program for generating and 
displaying CGs (computer graphics) is. received as auxiliary 
information 301. Then, by directly executing this CG 
program, CG video data can be displayed. Fig. 11 shows the 
data format of auxiliary information 3 01 handled by the 3 rd 
embodiment. As shown in the figure, an individual auxiliary 
information piece 304 comprises an auxiliary information 
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index 305 and a program or a script 1001. The auxiliary 
information index 305 includes a data type field 1002 
indicating the data type of the program or the script 1001. 
When executing a program or a script 1001, the processor 102 
refers to the data type field 1002, adjusting the execution 
of the program or the script 1001 to the data type specified 
in the field 1002. 

Unlike an operation to play back video and audio data, 
the execution time of a program or a script much depends on 
the performance of the processor 102 carrying out processing 
and the complexity of a network including a data source 
indicated by a link specified as auxiliary information 301. 
It is thus difficult to estimate the length of the execution 
time. For this reason, the auxiliary information index 3 05 
includes a processing time field 1003 . A positive value set 
in this processing time field 1003 is an effective period 
of time during which a result of execution of the program 
or the script 1001 is to be displayed. As the period of time 
lapses, the operation of the broadcast signal receiving 
apparatus is automatically switched to an operation to play 
back video and audio data 2 02 of broadcast information 201 
of the program. On the other hand, a negative value set in 
this processing time field 1003 indicates that the execution 
of the program or the script 1001 is continued till an 
instruction to end the execution is issued by the viewer. 



-32- 



Thus, in the case of a positive value set in this processing 
time field 1003, the processor 102 keeps monitoring the 
execution time while the program or the script 1002 of 
auxiliary information 201 is being executed. By virtue of 
the 3 rd embodiment, it becomes possible to provide the viewer 
with auxiliary information 301 including computer graphics, 
numbers, texts and figures in addition to video data by using 
various kinds of media. 

Industrial Applicability 

The present invention makes a package media 
interlocked broadcasting / watching / listening system 
feasible. In the package media interlocked broadcasting / 
watching / listening system auxiliary information relevant 
to broadcast information is distributed by using for example 
package media and, by playing back and displaying the 
auxiliary information and the broadcast information 
synchronously, the broadcast information can be 
supplemented with the auxiliary information. 



